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Abstract  -  The  panel  session  presents  collaborative  work 
undertaken  between  Australia ,  Canada ,  the  United 
Kingdom,  and  the  United  States  on  higher-level 
information  fusion  through  The  Technical  Cooperation 
Program  panel  on  Information  Fusion.  The  session 
outlines  developments  that  have  been  undertaken  by  the 
different  nations  and  features  a  demonstration  of  different 
nation’s  products  running  collectively  on  the  Coalition 
Distributed  Information  Fusion  Testbed.  The  context  for 
interaction  is  a  scenario  which  the  panel  wishes  to  make 
available  as  a  benchmarking  scenario  to  the  fusion 
community. 

Keywords:  Higher-Level  Fusion,  Situation  Assessment, 
Impact  Assessment,  Situation  Awareness,  Situation 
Analysis. 

1  Introduction 

This  paper  is  concerned  with  the  theory  and  practical 
implementation  of  higher-level  fusion  systems.  It  relates 
to  a  collaborative  program  of  work  in  higher-level  fusion 
that  has  been  conducted  between  Australia,  Canada,  the 
United  States  and  the  United  Kingdom  through  The 
Technical  Cooperation  Program  (TTCP)  panel  on 
Information  Fusion.  The  paper  presents  contributions 
from  the  four  nations,  showing  how  the  work  has  been 
integrated  through  a  common  scenario,  which  the  panel 
proposes  to  subsequently  offer  as  a  benchmarking 
scenario  to  the  higher-level  fusion  community.  The  paper 
outlines  developments  that  have  been  undertaken  and  the 
panel  session  associated  with  the  paper  features  a 
demonstration  of  different  nation’s  products  running  on 
the  Coalition  Distributed  Information  Fusion  Testbed 
(CDIFT). 

1.1  Vision 

The  remit  of  the  TTCP  panel  on  Information  Fusion  is  to 
“...promote  collaborative  research  and  experimentation 
between  the  member  nations  in  the  area  of  Information 
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Fusion  ...”.  The  panel  has  embraced  a  broader  vision, 
however.  There  is  currently  a  tendency  for  data  fusion 
proponents  to  belong  to  either  a  lower-level  fusion  or 
higher-level  fusion  camp,  with  the  higher-level  fusion 
camp  operating  more  as  an  aggregation  than  as  a 
community.  The  demand  for  data  fusion  at  all  levels, 
meanwhile,  increases  at  a  considerable  pace  as  the  effects 
of  the  Information  Age  propagate.  In  response  the  TTCP 
panel  on  Information  Fusion  is  seeking  to  open  up  some 
its  work  and  export  its  strong  sense  of  community. 

1.2  Content 

The  TTCP  panel  on  Information  Fusion  has  established  a 
sense  of  community  by  collaborating  on  both  theoretical 
and  practical  pursuits.  The  publication  of  a  book  by  the 
panel  ([1])  reflects  the  theoretical  collaboration.  This 
paper  reports  on  practical  collaboration. 

Two  ingredients  have  been  essential  for  practical 
collaboration.  The  first  has  been  the  development  of  a 
common  scenario  as  a  context  for  collaboration.  This 
allows  participants  to  map  their  problems  and  solutions 
into  a  common  environment.  The  second  has  been  the 
construction  of  the  Coalition  Distributed  Information 
Fusion  Testbed  (CDIFT).  This  allows  products  from  the 
contributing  nations  to  be  exercised  concurrently.  Section 
2  outlines  the  common  scenario  while  section  3  presents 
the  CDIFT  architecture. 

The  remainder  of  the  paper  highlights  some  of  the 
contributions  from  the  different  nations.  Section  4 
summarises  Canadian  scenario  platforms,  sensor  models, 
trackers  and  early  higher-level  fusion  capabilities.  In 
section  5,  aspects  of  the  US  Air  Force  Research 
Laboratory  Fusion2+  product  are  presented.  This  is 
followed  in  section  6  by  an  overview  of  a  United 
Kingdom  clustering  approach.  Section  7  offers  some 
remarks  about  the  Australian  higher-level  fusion 
implementations,  followed  by  the  Australian  higher-level 
fusion  display  technology  in  section  8.  The  subsequent 
two  sections  highlight  vignettes  in  urban  operations  and 
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coalition  search  and  rescue  respectively  that  are  being 
developed  for  inclusion  into  the  scenario. 

2  Scenario 

This  section  presents  a  very  short  description  of  a  vignette 
([2])  designed  to  stimulate  and  test  high-level  information 
fusion  concepts  and  algorithms  being  studied  by  our 
TTCP  panel.  We  created  a  scenario  (context)  called 
Atlantis  in  which  vignettes  can  be  developed  to  represent 
various  defence  and  security  problems  (e.g.  military 
strikes,  combat  search  and  rescue  (CSAR),  urban 
operations,  cyber  security,  harbour  security,  et  cetera. 

Atlantis  is  a  fictitious  continent  located  in  the  North 
Atlantic  Ocean,  between  Europe  and  Greenland.  For  the 
purpose  of  this  scenario,  the  land  areas  of  Iceland,  Ireland 
and  UK,  Shetland  excepted,  do  not  exist.  The  shape  of  the 
continent  looks  like  the  USCON  rotated  by  90°  with  one 
third  of  its  size.  As  shown  in  Figure  1,  Atlantis  is 
composed  of  six  countries:  Blueland,  Orangeland, 
Redland,  Brownland,  Whiteland  and  Greyland.  The 
historical  background  to  the  crisis  is  summarized  in  [2]. 
Briefly,  in  the  nineteen  century,  most  of  Atlantis, 
Greyland  excluded,  formed  the  Radobecan  Empire  until 
the  end  of  World  War  I  when  was  divided  into  the  five 
current  countries.  The  change  of  the  borders  in  addition  to 
the  different  social,  political  and  economic  conditions 
have  brought  disagreements  between  the  countries. 


Figure  1:  Focation  of  the  Atlantis  countries. 

One  of  the  vignettes  that  we  have  developed  so  far  is 
centred  on  a  2- ship  convoy  being  simultaneously  attacked 
by  submarine  and  aircraft  off  the  Atlantis  west  coast.  The 
two  surprised  attacks  occur  while  other  activities  are 
taking  place  in  order  to  complicate  the  assessment  of  the 
situation.  Those  activities  include  commercial  air  traffic, 
maritime  traffic,  observation  of  whale  migration,  and 
other  military  operations  such  as  air  surveillance 
patrolling  and  missile  deployment.  Moreover,  one  of  the 
opponent  countries  has  planned  to  take  control  of  the 
Celtic  Straits  on  that  day.  This  vignette  called  ‘Military 
Strikes  in  Atlantis’  describes  specific  military  activities 


within  an  evolving  context  where  various  crises  happened 
between  the  countries  as  illustrated  in  Figure  2. 
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Figure  2:  Focation  of  the  main  conflicts  prior  to  the 
current  crisis. 

The  current  crisis  advanced  when  two  mysterious 
explosions  occurred  in  the  Manghalour  Peninsula.  The 
first  explosion  hit  one  oil  storage  tank  in  Fage  and  the 
second  one  destroyed  a  section  of  the  Fage-Baldrik 
pipeline.  Orangeland  widely  argued  that  Blueland  could 
no  longer  protect  the  oil  reserves  that  are  crucial  to  them 
from  terrorist  attacks.  A  week  later,  Orangeland  forces 
invaded  the  Manghalour  Peninsula,  taking  control  of  its 
airspace  and  moving  forward  surface  to  air  missiles 
(SAMs).  They  promptly  neutralized  Blueland’ s  air 
defence  systems  around  the  North  Bay  and  took  control  of 
the  Bay’s  entrance,  confining  several  of  Blueland’ s 
vessels  of  war,  especially  Fast  Patrol  Boat  (FPBs)  and 
Mine  Counter  Measure  (MCM)  vessels,  near  Breivik. 

The  UN  Security  Council  (UNSC)  progressively  issued  a 
set  of  resolutions  to  solve  the  crisis  in  northern  Atlantis. 
The  first  resolution  orders  an  immediate  cease-fire  and  the 
withdrawal  of  Orangeland  forces  from  Blueland.  Further 
to  the  refusal  of  Orangeland  to  comply  with  that 
resolution,  a  second  resolution  was  approved  requesting 
all  states  to  prevent  any  trade  with  Orangeland.  The  UN 
Security  Council  announced  a  third  resolution  imposing 
an  embargo  on  Orangeland  and  requested  the  Alliance 
Council  for  military  assistance  to  restore  international 
peace  and  security  in  northern  Atlantis.  Two  North 
America  countries  decided  to  participate  by  sending  a  task 
group  to  enforce  that  resolution.  The  task  group, 
composed  of  1  x  Wasp  class  FHD,  1  x  Burke  class  DDG, 
1  x  Iroquois  class  DDG  and  2  x  Halifax  class  FFG,  have 
left  their  home  base  toward  the  North  Atlantic  Ocean  and 
Rockall  Sea. 

The  vignette  is  composed  of  various  actions  or  activities 
occurring  in  the  same  time  frame.  Those  components 
include  the  following. 
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Redland  warships.  Description  of  the  operations  carried 
out  by  the  Redland  Navy  in  support  of  the  planned  air 
attack  of  the  Celtic  Straits.  Those  operations  include  the 
dispatching  of  vessels  in  the  Atlantic  Ocean  off  the  Celtic 
Straits,  the  boarding  of  merchant  ships  going  to  Blueland 
seaports  in  the  Celtic  Sea,  and  the  escort  of  merchant 
ships  from  Redland  seaports  through  the  Celtic  Straits. 

Alliance  Convoy.  Description  of  the  convoy  composed  of 
the  cargo  and  Alliance  Task  Group  leaving  North 
America  in  destination  to  Atlantis. 

Commercial  Air  corridors.  Description  of  the  air  corridors 
used  by  commercial  airplanes  for  the  transoceanic  flights 
between  Europe  and  North  America  and  the  domestic 
flights  in  the  Atlantis  continent. 

Blueland  Ground-Based  Radars.  Description  of  the  radar 
network  used  by  Blueland  Air  Force  for  air  surveillance 
and  air  traffic  control. 

Maritime  Routes.  Description  of  the  commercial  maritime 
routes  usually  taken  by  the  merchant  ships  going  to  or 
coming  from  the  Celtic  Straits. 

Whale  Migration.  Description  of  the  platforms  watching 
the  humpback  whales  swimming  off  the  Atlantis  western 
coast  towards  the  North  Atlantic  Ocean. 

Missiles  Deployment.  Description  of  the  missiles 
previously  deployed  by  Redland  in  support  of  the  air 
attack  of  the  Celtic  Straits. 

Redland  Airborne  Surveillance.  Description  of  the  air 
platform  used  by  Redland  to  detect  and  track  operations 
carried  out  beyond  its  border. 

Submarine  Attack.  Description  of  the  attack  of  the  convoy 
by  an  Orangeland  submarine. 

Air  Attack  of  the  Convoy.  Description  of  the  attack  of  the 
convoy  by  2  Redland  fighters. 

Air  Attack  of  the  Celtic  Straits.  Description  of  the  fighters 
sent  by  Blueland  to  counter-attack  the  Redland  fighters 
and  the  attack  performed  by  Redland  to  take  control  of  the 
Celtic  Straits.  This  second  attack  involves  air,  navy  and 
ground  platforms. 

3  CDIFT 

The  Coalition  Distributed  Information  Fusion  Testbed 
(CDIFT)  is  an  initiative  of  the  TTCP  panel  to  establish  a 
distributed,  heterogeneous  coalition  environment  to 
support  development  and  evaluation  of  information  fusion 
technologies  and  applications. 

The  CDIFT  incorporates  a  synchronous  simulation  layer 
using  HLA  to  support  ground  truth  and  sensor  modeling, 
and  an  asynchronous  notional  ’’Joint  Task  Force 
Information  Grid”  (JTFIG)  to  represent  real-time  access  to 
coalition  information  sources  and  sensor  feeds.  The  US 
Joint  Battlespace  Infosphere  (JBI)  is  used  as  the  primary 
medium  for  information  exchange  between  coalition 
systems  via  a  publish/subscribe  model.  Other  technologies 


are  also  exploited,  such  as  web  services,  and  the 
Australian  open-source  Avis1  architecture  as  an  agent 
messaging  layer.  Applications  access  these  layers  as  their 
information  requirements  dictate. 


Figure  3:  Architecture  employed  in  the  Coalition 

Distributed  Information  Fusion  Testbed  (CDIFT)  provides 

a  distributed,  heterogenous  coalition  environment. 

The  functional  components  of  the  CDIFT  follow. 

1 .  Ground-truth  generation  is  done  using  STAGE, 
which  can  inject  ground- truth  into  the  scenario  in 
real-time,  or  be  saved  as  data  files  for  later  playback. 

2.  Sensor  modeling  and  tracking  provides  sensor  and 
track  reports  that  are  published  onto  the  JTFIG  and 
available  to  any  application  that  subscribes  to  these 
reports.  Both  ‘cookie-cutter’  and  realistic  high- 
fidelity  sensor  and  tracker  models  are  available  on  the 
CDIFT. 

3.  Information  fusion  applications  and  services 
subscribe  to  the  sensor  and  tracker  reports,  and 
publish  complete  and/or  partial  information  fusion 
products  back  on  the  JTFIG  so  as  to  be  available  to 
other  applications. 

4.  Point-to-point  communication  between  applications, 
subcomponents,  or  agents  when  processing 
information  utilizes  the  agent  messaging  layer.  This 
provides  a  level  of  abstraction  that  allows  brokers  to 
locate  agents  with  the  required  capabilities,  and 
multiple  agents  to  negotiate  an  appropriate  solution  as 
the  system  complexity  increases.  The  CDIFT  is 
exposed  as  a  web  service  using  a  web  proxy.  Agents 
register  their  capabilities  with  the  web  proxy,  which 
then  tasks  them  to  satisfy  web  service  requests. 

5.  User  interfaces  are  provided  by  individual 
applications,  and  also  by  report  subscriber  services  on 
the  JTFIG.  These  can  be  deployed  at  whatever  sites 
they  are  needed,  and  configured  to  only  subscribe  to 
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reports  that  are  of  interest  to  the  particular  site,  role, 
or  context.  The  way  the  information  is  presented  to 
the  users  is  also  configurable  to  support  different 
roles,  contexts,  and  user  preferences. 

4  Platforms,  Sensor  Models,  Trackers 

The  numerous  agents  (sensors,  platforms,  weapons, 
organisations,  et  cetera )  of  the  vignette  components  listed 
in  section  2  (Redland  warships,  Alliance  Convoy 
Commercial  Air  corridors,  Blueland  Ground-Based 
Radars,  Maritime  Routes,  Whale  Migration,  Missiles 
Deployment  Redland,  Airborne  Surveillance,  Submarine 
Attack,  Air  Attack  of  the  Convoy,  Air  Attack  of  the  Celtic 
Straits)  are  exercised  through  high-level  information 
fusion  applications  on  the  CDIFT  infrastructure  of  section 
3,  to  help  tell  a  story  about  what  is  going  on  in  Atlantis. 

Detailed  simulations  are  being  used  to  represent  sensors 
(e.g.  radars)  and  platforms.  We  attempt  to  simulate  level  1 
data  fusion  object  assessments  (tracks  and  id)  since  the 
object  of  our  investigations  is  high-level  information 
fusion.  Below  are  short  excerpts  of  the  way  each 
component  of  the  vignette  is  being  simulated.  All  details 
are  described  in  [2].  For  the  purpose  of  illustration  only,  a 
couple  of  examples  are  outlined  here. 

4.1  Redland  Warships 


4.3  Commercial  Air  Corridors 

Two  types  of  air  routes  are  considered  in  this  vignette:  the 
North  Atlantic  air  routes  used  for  commercial  flights 
between  North  America  and  Europe,  and  the  Continental 
Atlantis  air  routes  for  air  service  between  Atlantis 
airports.  To  make  connections  between  European  and 
North  American  eastern  cities,  airplanes  have  to  fly  over 
the  northern  Atlantic  Ocean,  and  consequently  the  North 
Sea  and  the  Atlantis  continent,  whereas  European  and 
North  American  western  cities  are  usually  linked  through 
the  Arctic.  Figure  4  shows  typical  air  routes  used  by 
airplanes  to  cross  the  northern  Atlantic  Ocean. 


Figure  4:  Typical  routes  of  the  North  Atlantic  Airspace. 


On  their  way  to  carry  out  naval  exercises  in  the  Atlantic 
Ocean,  Redland  vessels  of  war  —  1  x  FFG  (Kotor  class) 
and  2  x  PCF  (Rafael  class:  ex- Yugoslavia  Koncar  class) 
—  have  reported  detecting  submarine  activities  and  found 
moored  mines  off  the  Celtic  Straits.  Redland  claimed  that 
the  straits  were  no  longer  safe  for  the  transit  of  merchant 
ships  because  of  the  likely  presence  of  mines  or 
submarines,  and  blamed  Blueland  for  its  conflict  with 
Orangeland  as  being  the  main  cause  of  the  hazard  in  the 
straits.  For  that  reason,  Redland  sent  2  x  MCM  (Birt  class: 
ex- Yugoslavia  Klanac  class)  and  1  x  SSK  (Elwood  class: 
ex-NL  Walrus  class)  to  search  for  mines  and  submarines. 

4.2  Convoy 

A  cargo  ship  carrying  ammunition  has  just  left  a  North 
America  seaport  in  the  direction  of  Bercelport  (Celtic  Sea) 
in  Blueland,  while  the  task  group  is  going  to  the  Rockall 
Sea  (north  of  Atlantis).  To  protect  the  cargo  from  being 
boarded  by  Redland  vessels,  Blueland  has  requested  that 
the  task  group  escort  her  until  they  meet  a  Blueland  Navy 
ship  (lx  Descubierta  class  FFG)  off  the  Celtic  Straits. 
The  frigate  has  deployed  her  towed  array  sonar  and  her 
CH-124  (Sea  King  Helicopter)  ready  to  detect  and  hunt 
submarines.  The  frigate  sails  at  15  knots,  1  NM.  ahead  of 
the  cargo.  The  helicopter  is  equipped  with  ping  sonar  and 
2  x  Mk-32  torpedoes. 


4.4  Blueland  Ground-Based  Radars 


Figure  5:  Location  and  coverage  of  the  Blueland  long- 
range  and  short-range  radars. 

Blueland  airspace  is  divided  into  three  Air  Defence 
Sectors  (North-East,  North-West  and  South)  as  per  Figure 
5.  The  Operations  Centre  of  each  sector  controls  all  flying 
activities  (military,  commercial  and  civilian)  in  the  sector. 
The  South  Sector,  which  is  in  charge  of  the  Celtic  Straits 
area,  is  bounded  to  the  south  by  the  Brownland  and 
Redland  borders,  to  the  west  by  the  Atlantic  Ocean,  and  to 
the  north  by  the  following  segment  points:  58°30’N 
27°00’W,  58°30’N  25°00’W,  58°00’N  21°00’W,  and  the 
junction  of  the  borders  with  Redland  and  Orangeland 
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(57°00'N  18°00’W).  The  Operation  Centre  of  the  South 
Sector  is  located  at  Nellis  air  force  base  (AFB)  (57°06'N 
22°22’W). 

4.5  Events  and  ORBAT 

A  similar  description  applies  for  other  components  of  the 
vignette  ([2]).  The  exact  time  sequence  of  all  events  is  in 
the  simulation  environment  of  CDIFT.  In  addition,  each 
specific  entity,  such  as  a  frigate,  is  simulated  according  to 
order  of  battle  (ORBAT)  information  like  that  featured  in 
the  figure  below. 


FFG 

Sensors: 

Halifax 

-Air  search  radar:  SPS-49  (C/D-band;  457  km/250  NM;  h:~65’) 

-Air/surface  search  radar:  Sea  Giraffe  (G/H-band;  100  km/55  NM;  h:~85’) 

-Fire  control  radar:  2  x  STIR  1 .8  (l/K-band;  66  km/  36  NM  for  1  m2  target) 

-Navigation  radar:  l-band. 

-Hull-mounted  sonar:  SQS-51 0  (active  search  and  attack;  medium  frequency.) 

-Towed  array  sonar:  SQR-501 

Weapons: 

-SSM:  8x  Harpoon  (active  radar  homing;  130  km/70  NM  at  0.9  M) 

-SAM:  16  x  Sea  Sparrow  (semi-active  radar  homing;  14.6  km/8  NM  at  2.5  M) 

-Gun:  1  x  Bofors  57  mm  (220  rds/min;  1 7  km/9  NM;  weight  of  shell  2.4  kg) 

-Gun:  1  x20mm  Phalanx  (anti-missile;  3,000  rds/min;  1.5  km) 

-Gun:  8  x  1 2.7  mm  Machineguns. 

-Torpedo:  4  x  Mk32  (24  x  Mk46  Mod  5;  anti-sub;  active/passive  homing;  1 1  km/5.9  NM;  40  kt) 

Countermeasures: 

-Decoys:  P8  chaff,  P6  IR  flares,  Nixie  SLQ-25  towed  acoustic  decoy. 

-ESM:  Canews  SLQ-501  (radar  warning/intercept;  0.5-18  GHz) 

-ECM:  Ramses  SLQ-503  (jammer). 

Aircrafts: 

-Helicopters:  1  x  Sea  King  (CH-124A  ASW  or  CH-124B  Heltas) (125  knots;  330  NM;  Mk-32 

torpedoes) 

Figure  6:  Sample  from  the  Scenario  ORBAT 

5  Fusion  2+ 


The  US  Air  Force  Research  Laboratory  (AFRL) 
contribution  to  the  CDIFT  Testbed  is  the  Joint  Battlespace 
Info  sphere  (JBI)  that  is  being  utilized  as  the 
publish/subscribe  environment  by  the  other  nations;  along 
with  the  Fusion  2+  Testbed  which  will  provide  software 
tools  for  processing  the  text  reports  and  analyzing  them  to 
determine  the  current  situation.  The  Fusion  2+  testbed  has 
been  developed  to  support  information  fusion  research 
and  development.  As  first  discussed  in  [3]  and  [4],  the 
Fusion  2+  testbed  is  comprised  of  the  following 
components: 

1 .  Data  collection 

2.  Document  Parsing 

3.  Model  Analysis 

Initially,  the  analyst  will  specify  a  region  of  interest,  and 
identify  specific  items  of  interest,  which  will  be  used 
develop  (or  modify)  models.  These  models  will  then  be 
used  to  drive  the  data  collection. 

The  Data  collection  is  performed  via  an  AFRL  developed 
meta-search  engine  that  can  simultaneously  query  and 
retrieve  documents  from  multiple  sources.  These 
documents  are  then  distributed  for  parsing. 

Natural  Language  Extractors  are  utilized  to  parse  free  text 
messages  and/or  documents.  Formatted  messages,  such  as 
Tactical  Reports  (TACREPs),  are  processed  by  the 
Generic  Intelligence  Processor  (GIP)  ([5]). 


Model  Analysis  tools  then  utilize  the  evidence  provided  to 
ascertain  if  any  segment  of  a  model  is  unfolding.  The 
model  analysis  tool  leverages  graph  theory,  and  searches 
the  input  graph  (which  is  generated  based  on  the 
evidence)  for  any  subset  of  the  target  graphs  (which  are 
developed  a  priori  by  the  analyst).  Graph  matches  which 
exceed  a  specified  threshold,  are  provided  as  potential 
alerts  to  the  analyst. 

The  Fusion  2+  testbed,  as  envisioned  for  use  in  CDIFT, 
will  require  two  modifications.  First  is  the  development 
of  synthetic  "open-source"  documents  based  on  the  North 
Atlantis  scenario  vignette  that  will  be  used  as  the  inputs  to 
the  Fusion  2+  software.  These  documents  will  either  be  in 
the  form  of  TACREPs,  or  news  reports.  The  other 
modification  entails  the  development  of  new  models  for 
the  Model  Analysis  tool.  Once  again,  these  models  are  to 
be  specified  by  the  analyst  based  on  their  experience  with 
the  area  of  interest.  Both  of  these  modifications  are  a 
work  in  progress. 

6  Indicators  of  Collective  Behaviour 

The  UK’s  Indicators  of  Collective  Behaviour  (ICB) 
algorithm  exemplifies  the  potential  that  CDIFT  offers  as 
an  environment  for  evaluating  fusion  algorithms.  ICB  was 
in  fact  not  designed  to  be  an  operational  tool,  but  rather  as 
a  training  aid  for  future  commanders  to  reveal  the  sorts  of 
future  capability  that  would  be  enabled  by  fusion.  For  this 
role,  the  ICB  algorithm  was  developed  for  Dstl’s 
Wargame  Infrastructure  and  Simulation  Environment 
(WISE)  facility  which  is  a  formation  level,  land-based 
wargame  used  as  an  operational  analysis  model  ([6]). 

WISE  can  operate  both  as  a  wargame  in  which  military 
players  command  simulated  forces  interacting  in  a 
simulated  environment  and  as  a  simulation  which  operates 
without  any  human  interaction.  In  a  typical  wargame, 
WISE  might  portray  a  complex  warfighting  environment 
with  hundreds  of  manoeuvring  entities  under  the  control 
of  “Red”  and  “Blue”  commanders,  each  of  whom 
responds  to  his  perception  of  the  situation  which  in  turn  is 
generated  by  the  (simulated)  ISTAR  entities  under  his 
control.  In  general,  each  commander’s  view  of  the 
battlefield  is  both  imprecise  (because  there  is  a  stochastic 
element  in  WISE’s  sensor  models)  and  incomplete 
(because  of  terrain,  weather  and  sensor  range  effects).  But 
the  application  ICB  offsets  some  of  these  effects,  leading 
to  improved  situational  awareness  and  (at  least  in 
principle)  measurably  improved  military  outcome. 

6.1  ICB  Algorithm 

The  proposition  underlying  the  ICB  algorithm  is  that 
manoeuvring  entities  which  are  close  together  and/or 
moving  at  similar  speeds  and/or  moving  in  similar 
directions  are  likely  to  be  acting  collectively  and  as  such, 
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might  be  posing  a  threat  to  a  headquarters  (HQ)  or  other 
deployed  asset  upon  which  they  appear  to  be  advancing. 
The  algorithm  has  been  fully  described  in  [7],  and  it  has 
three  elements: 

•  Identification  of  candidate  clusters; 

•  Assessing  confidence;  and 

•  Inferring  intent. 

6.2  Identifying  Candidate  Clusters 

Assuming  a  total  of  N  battlespace  entities  have  been 
detected  and  tracked  (either  in  the  CDIFT  or  in  the  WISE 
wargame)  and  that  their  positions,  speeds  and  headings 
are  known  (possibly  with  errors),  a  fully-connected  bi¬ 
directional  weighted  graph  of  these  entities  may  then  be 
drawn  up  in  which  the  edge  weights  (EWy)  represent 
dimensionless  assessments  of  separation,  speed  and 
heading: 


each  cluster.  For  WISE  and  CDIFT  a  simpler  approach 
has  been  adopted,  based  on  the  (instantaneous)  similarity 
of  speed  and  heading  between  group  members.  The  basic 
idea  here  is  that  a  candidate  group  whose  members 
display  a  “small”  standard  deviation  in  speed  or  heading 
is  more  likely  to  be  genuine  than  one  with  a  “large” 
standard  deviation.  Normalising  by  range,  and  taking  due 
care  to  avoid  numerical  artefacts  associated  with  the 
apparent  discontinuity  between  0°and  360°  in  heading 
gives  a  measure  Confidencec  defined  as  follows: 


Confidencec  = 


Px°(sitj) 

vmax{y..}-min{y.J} 


rx<r(Kj) 

max {ht  j}- min {hi  . } 


Different  confidence  values  may  be  used  by  commanders 
in  deciding  the  priority  order  in  which  apparent  threats 
will  be  addressed. 
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where  dy  is  the  distance  between  entities  i  and  j,  Asy  and 
Ahy  are  differences  in  their  speed  and  heading  and  a,|3  and 
y  are  constants  which  can  be  determined  dynamically  at 
run  time  to  give  greater  or  lesser  emphasis  to  separation, 
speed  and  heading  criteria.  Once  the  graph  has  been 
assembled,  the  Minimum  Spanning  Tree  (MST)  is 
calculated  using  Dijkstra’s  method  and  this  is 
subsequently  broken  down  into  “forests”  using  a  Gaussian 
Parzen  Window  estimator  to  identify  edges  which  should 
be  cut.  The  result  is  a  set  of  candidate  clusters  (Figure  7). 


Figure  7:  Example  of  an  8  node  graph  showing  the 
Minimum  Spanning  Tree  and  subsequent  allocation  into 
candidate  clusters. 


6.3  Assessing  Confidence 

Once  candidate  clusters  have  been  identified  they  can  be 
monitored  and  in  an  operational  system  an  analysis  of 
cluster  parameters  over  time  would  probably  be  the  best 
way  to  assign  measures  of  confidence  in  the  coherence  of 


6.4  Inferring  Intent 

The  point  of  CDIFT  is  that  a  variety  of  different 
information  sources,  including  kinematic  criteria  and 
natural  language  reports,  may  be  fused  in  order  to  infer 
intent.  Within  the  ICB  algorithm  an  initial  indicator  of 
intent  is  provided  by  a  simple  analysis  of  the  overall 
direction  that  a  group  is  taking  in  relation  to  known 
targets  (in  this  case  the  locations  of  important  Blue 
entities,  such  as  HQs).  A  pragmatic  approach  for  this  has 
been  developed  that  is  quick  and  easy  to  compute,  based 
on  the  angular  difference  in  bearing,  0,  between  a  target 
and  the  mean  heading  of  a  group.  This  is  called  a  “Target 
Commitment  Function”,  x(0),  and  it  is  defined  as 

t(0)=O.  5  [  1  +cos2n+I  (0)] 

where  0  is  the  difference  in  heading  between  that  of 
designated  target  and  the  mean  heading  of  the  cluster  and 
n  is  a  free  parameter,  designed  to  give  an  appropriately 
shaped  functional  form  (in  initial  work  n  was  set  to  a 
value  of  10). 


Target  Commitment  Function 


Figure  8:  Target  Commitment  function  for  n=10. 

The  advantage  of  this  formulation  is  that  x  lies  between  0 
and  1  and  it  is  essentially  tri-valued,  with  x=l  indicating  a 
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cluster  that  is  moving  towards  a  target,  x=0  one  that  is 
moving  away  and  x=0.5  representing  the  undecided  case 
(Fig  2).  Exact  values  of  x  may  be  used  to  help  Blue 
commanders  prioritise  between  different  threats  and  make 
appropriate  force  allocations. 

6.5  CDIFT  application 

In  the  North  Atlantis  “Military  Strikes”  scenario  there  are 
a  number  of  entities  that  are  genuinely  moving  together 
such  as  the  FI 6  fighters  and  shipping  convoy.  These  are 
readily  identified  by  ICB,  giving  confidence  that  more 
complex  behaviours  expected  in  other  CDIFT  vignettes 
will  also  be  correctly  discovered  and  at  the  same  time 
providing  a  diverse  set  of  situations  within  which  the  ICB 
confidence  and  intent  indicators  and  other  parameters  may 
be  studied. 


registration  at  level  2;  and  situation  assessment  at 
level  3. 


World 


The  successful  transition  from  a  land-based  training  aid  to 
a  marine-based  “operational”  tool  has  demonstrated  the 
flexibility  of  the  CDIFT  testbed  as  well  as  providing 
valuable  opportunities  for  refining  the  ICB  algorithm. 

7  STDF  Model 

The  Australian  State  Transition  Data  Fusion  (STDF) 
Model  ([8], [9], [10])  was  developed  as  a  unifying  model 
across  the  JDL  levels  of  fusion  ([11]). 


Figure  10:  STDF  Model. 

This  section  of  the  paper  focuses  on  implementation  of 
the  STDF  model  for  higher-level  fusion  in  the  context  of 
the  North  Atlantis  scenario. 

7.1  State  Representation 

Figure  1 1  presents  the  STDF  Model  for  level  2  situation 
assessments. 


Figure  9:  The  JDL  Model  of  Data  Fusion 

The  STDF  Model  is  based  on  two  premises. 

1 .  At  each  level  of  fusion  the  world  can  be  assessed  in 
terms  of  states  and  transitions  between  those  states. 
For  example,  level  1  states  are  state  vectors ;  level  2 
states  are  states  of  affairs ;  level  3  states  are  scenario 
states ;  with  objects ,  situations  and  scenarios  being 
the  sets  of  respective  transited  states  over  time. 

2.  At  each  level  of  fusion,  the  fusion  process  adheres  to 
the  same  pattern  of  behaviour  but  the  nature  of  the 
content  changes.  Figure  10  illustrates  the  common 
pattern  of  behaviour.  For  example,  registration  is 
coordinate  registration  at  level  1 ;  semantic 
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Figure  11:  STDF  Model  for  Situation  Assessments 

At  level  2,  the  world  is  understood  in  terms  of  situations 
as  transitions  between  states  of  affairs,  with  states  of 
affairs  expressed  as  a  set  of  statements  about  the  world  in 
a  formal  language  associated  with  a  formal  logic.  The 
inclusion  of  a  formal  logic  allows  the  semantics  of  the 
interpretable  symbols  in  the  formal  language  to  be 
defined,  or  at  the  very  least  constrained.  The  choice  of 
interpretable  symbols  in  the  formal  language  depends 
upon  the  nature  of  the  domain  under  consideration. 
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For  the  military  oriented  North  Atlantis  vignette,  the 
Mephisto  framework  ([12], [13], [10])  has  been  used  to 
identify  the  basic  interpretable  symbols.  Mephisto 
characterizes  the  world  through  5  tiers  and  identifies 
concepts  and  associated  interpretable  symbols  for  each 
tier.  Military  events  in  the  world  typically  involve  all  five 
tiers  concurrently. 

Social:  ally,  agrees,  possesses,  commands,  ... 

Cognitive:  believes,  expects,  prefers,  perceives,  ... 

Functional:  senses,  strikes,  informs,  moves,  ... 

Environmental:  air,  water,  upland,  outer_space,  ... 

Metaphysical:  exists,  identical,  before,  connects,  ... 
Logical  constraints  specify  the  meaning  of  each  of  these 
terms.  Implementation  of  these  logical  constraints  then 
constrains  the  machine’s  interpretation  of  those  concepts. 
Some  implemented  examples  of  logical  constraints 
include: 

identical(X,  Y)  if  fragment(X,  Y)  &  fragment(Y,  X). 
connects(X,  X)  if  exists(X). 
agrees(@(X,  T,  SI),  @(Y,  T,  S2),  Prop)  if 
offers(@(Y,  T1 ,  _),  @(X,  T1 ,  _),  Prop)  & 
intends(@(X,  T,  SI),  Prop)  & 
informs(@(X,  T,  SI),  @(Y,  T,  S2)  & 
intends(@(X,  T,  SI),  Prop))  & 
before(T1,  T). 

The  semantic  constraints  facilitate  both  numerical 
calculation  and  abstract  symbolic  reasoning.  The 
distance(C1,  C2,  D)  predicate  will  calculate  the  great  circle 
distance  D  between  coordinates  Cl  and  C2.  Knowing 
before(T1,  T2)  and  before(T2,  T3)  is  sufficient  to  deduce 
before(T  1 ,  T3),  even  when  numerical  times  for  T1 ,  T2  and 
T3  are  not  known. 

7.2  Observation 

At  level  2,  observation  comprises  object  assessment, 
semantic  registration  and  propositional  association 
(Figure  11).  In  the  North  Atlantis  context,  object 
assessments  are  the  tracker  outputs  described  in  section  4. 
These  are  typically  vectors  of  the  form 

<id,  time,  <x,  y,  vx,  vy>,  lat  ref,  long_ref,  P,  type> 
where:  id  is  a  unique  identifier;  time  is  a  relative  time;  x  is 
position  in  the  x  direction;  y  is  position  in  the  y  direction; 
vx  is  speed  in  the  x  direction;  vy  is  speed  in  the  y  direction; 
lat  ref  and  long_ref  are  the  relative  source  reference 
coordinates;  P  is  a  covariance  matrix;  and  type  is  a 
classification  between  air,  surface,  subsurface  or 
unknown.  The  following  is  a  sample  object  assessment 
vector. 

<t_821, 7200665,  <40193.1,-108826,  -215.141 , 209.048>, 
1.004414,  -0.465357, 

<<30627. 3,  -4925.19,  1326.65,  -154.695>, 

<-4925.19,  41766.1,-154.19,  1676.42>, 

<1326.65,  -154.19,  116.128,  -6.58736>, 

<-154.695,  1676.42,  -6.58736,  130.768»,  0>. 

The  semantic  registration  process  accepts  object 
assessments  and  translates  them  into  formal  sentences 
supported  by  the  Mephisto  framework.  The  following  is 


the  semantic  registration  output  of  the  previous  sample 
vector. 

at(t_821  ,timestamp(2001 ,6,16,13.0,45.0,51 .28), 
coordinate(radians(0. 9873294202374645), 
radians(0. 4539469822836761 ), 
(metres(0.0),metres(1 00000.0)))), 
speed  (@(t_821  ,timestamp(2001 ,6,16,13.0,45.0,51 .28),_1 865), 
metres_per_second(299. 977859491 3298)), 
course(@(t_821,timestamp(2001,6,16,13.0,45.0,51.28),_1880), 
radians(2.341 831 597675561 5)), 
tell_in_air(@(t_821  ,timestamp(2001 ,6,16,1 3.0,45.0,51 .28), 
coordinate(radians(0. 9873294202374645), 
radians(-0. 4539469822836761 ), 
metres(5.0E+04))), 
celtic_sea_ext*redland_region), 

unknown_allegiance(@(t_821  ,timestamp(2001 ,6,16,13.0,45.0,51 . 
28), _ 331 8)), 

[[40193.1,-108826.0,-215.141,209.048], 

[4021 6.76791 9900805,-1 09061 .98584052833,- 
21 3.9553952891 1 1 04,204.80762301 955747],  . . .] 

This  registers  the  level  1  object  assessment  in  a  form  that 
can  be  reasoned  with  by  level  2  processes.  If  the  user 
wishes  to  receive  updates  on  any  semantically  registered 
information,  then  the  natural  language  generation  code 
can  be  applied.  When  applied  to  the  previous  example,  it 
generates  the  following  output. 

With  some  degree  of  uncertainty,  at  the  time  of  13.0  hours  45.0 
minutes  and  51.28  seconds  on  the  16th  day  of  June  2001,  t_821 
is  at  location  56.56  degrees  latitude,  -26.0  degrees  longitude,  with 
an  altitude  between  0.0  metres  and  100000.0  metres.  It  has  a 
speed  of  299.97  metres  per  second,  has  a  course  of  134.17 
degrees,  is  of  unknown  allegiance,  and  is  in  the  airspace  over 
Redland’s  Celtic  Sea. 

English  accounts  of  semantic  registrations  have 
previously  been  disseminated  to  the  users  through  virtual 
advisers  (see  section  8). 

Semantic  registration  is  a  value  adding  process  that  draws 
on  considerable  domain  knowledge  couched  in  Mephisto 
terms.  Domain  knowledge  covers  geography  (both  in 
abstract  region  connection  calculus  terms  and  through 
access  to  a  GIS  system),  the  capability  and  disposition  of 
available  forces,  the  political  alliances,  political  intent, 
and  the  like. 

Under  the  cognitive  model  outlined  in  section  7.3  below, 
semantically  registered  observations  are  classified  as 
perceptions.  The  propositional  association  challenge  is  to 
reconcile  the  incoming  perceptions  with  expectations  and 
anticipations  formed  through  the  prediction  process. 
Propositional  association  superficially  mirrors  level  1  data 
association  in  that  it  involves  gating,  prediction,  scoring 
and  assignment.  Gating  is  used  to  hash  candidate 
perceptions  into  different  clusters,  but  extends  beyond 
data  association  gating  by  providing  both  a  logical  and 
regional  basis  for  gating.  The  prediction  process  gathers 
the  candidate  expectations  and  anticipations.  In  low 
uncertainty  contexts  the  scoring  and  assignment  steps 
reduce  to  unification  based  pattern  matching.  In  higher 
uncertainty  environments,  in  which  multiple  perception  to 
expectation  associations  are  possible,  a  scoring  process  is 
used  to  rate  possible  associations  before  the  assignment 
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process  selects  pairings,  while  possibly  allowing  multi¬ 
hypothesis  options  to  proceed.  Mathematics  for  this  can 
be  found  in  [9]. 

7.3  Prediction  and  Explanation 

The  higher-level  STDF  process  is  being  implemented 
within  the  ATTITUDE  TOO  cognitive  architecture,  which 
has  recently  been  written  as  a  successor  to  the  Attitude 
cognitive  architecture  ([14]).  An  Attitude  Too  agent’s 
long-term  memory  comprises:  assertional  memory 
consisting  of  semantic  and  epistemic  long-term  memory, 
and  episodic  long-term  memory.  The  semantic  memory 
holds  the  logical  constraints  discussed  in  section  7.1.  The 
epistemic  memory  contains  the  declarative  domain 
knowledge  alluded  to  in  section  7.2.  Episodic  memory 
consists  of  cognitive  routines,  each  comprising  a  goal  and 
a  behavioural  recipe  for  achieving  that  goal,  expressed  as 
a  network  of  propositional  attitude  instructions.  Semantic 
memory  delivers  meaning;  epistemic  memory  delivers  the 
“know  that”  domain  knowledge  and  episodic  memory 
delivers  the  “know  how”  domain  knowledge. 

The  prediction  and  explanation  steps  are  primarily 
managed  through  cognitive  routines,  supplemented  by  the 
assertional  memory.  A  simplified  example  of  a  cognitive 
routine  for  monitoring  a  ship  traversing  a  sea  lane  is 
featured  below. 

routine(traversing_sea_lane), 

A(believe(i,  entering_sea_lane), 

(Number_Missed_Updates  is  0), 

(Max_Missed_Updates  is  3), 

*(A((Number_Missed_Updates  =<  Max_Missed_Updates), 
add_time(When,  timeperiod(0.0,0.0,0.0,1 .0),  Expiry), 
line_segment(sea_lane(Lane),  Terminal), 

+(A(expect(i, 

one_of([  updated_along_sea_lane, 
updated_across_sea_lane, 
updated_stopped_on_sea_lane, 
updated_enter_sea_lane_nexus]),  by(Expiry)), 
+(A(not_believe(i,  updated_enter_sea_lane_nexus), 
intend(i,  traversing_sea_lane_nexus,  priority(0.9)), 
disapprove), 

A(not_believe(i,  updated_along_sea_lane), 
desire(i,  updated_along_sea_lane)), 

A(not_believe(i,  updated_across_sea_lane) 
desire(i,  update_across_sea_lane)), 

A(not_believe(i,  updated_stopped_on_sea_lane), 
desire(i,  update_stopped_on_sea_lane_nexus)))), 
(Number_Missed_Updates 

is  Number_Missed_Updates  +  1)), 

When  is  New_When)), 

+(A((Number_Missed_Updates  >  Max_Missed_Updates), 
believe(i,  exited_sea_lane)), 
succeed)). 

The  example  routine  monitors  transitions  between 
entering_sea_lane,  along_sea_lane,  across_sea_lane, 
stopped_on_sea_lane  and  enter_sea_lane_nexus  states  of 
affairs.  The  routine  illustrates  expectations  being  set  for 
the  last  four  of  these  states  of  affairs  during  execution  and 
these  are  reconciled  with  perceptions  using  the 
proposition  association  process  discussed  in  section  7.2. 
This  particular  example  uses  uncertainty  to  perform  the 


association  process,  but  does  not  carry  the  uncertainty  into 
the  routine  and  does  not  pursue  a  multi-hypothesis 
approach,  though  both  can  be  done.  The  expectation 
signifies  the  prediction  process  while  the  beliefs  represent 
the  explanation  process.  In  that  respect  the  routine  is 
particularly  simple.  Higher  order  routines,  such  as 
predicting  the  likely  final  destination  and  time  of  arrival, 
draw  on  lower  level  routines  like  the  one  illustrated. 
Cognitive  routines  also  typically  deal  with  the  interacting 
behaviour  of  multiple  objects,  not  just  a  single  object  as  in 
the  simple  illustration. 

8  Higher  COP 

The  Common  Operating  Picture  (COP)  is  widely  used  to 
support  situational  awareness,  providing  a  2D  Mots  on 
maps’  display  that  shows  where  entities  are  located  in  the 
battlespace  relative  to  various  geospatial  features  -  i.e.  the 
outputs  of  lower-level  fusion.  However,  the  COP  does  not 
support  the  comprehension  and  projection  aspects  of 
situational  awareness  -  the  outputs  of  higher-level  fusion 
systems.  The  COP  leaves  the  cognitive  load  on  the  user  to 
interpret  the  ‘picture’  displayed.  Furthermore,  it  provides 
no  aid  to  achieving  ‘shared’  situational  awareness  for 
users  in  diverse  roles  and  operating  domains  -  how  they 
interpret  a  common  ‘picture’  will  be  influenced  by  their 
individual  context.  While  the  COP  may  be  sufficient  for 
lower-level  fusion,  a  context-sensitive  ‘Higher  COP’  or 
‘HiCOP’  is  needed  to  display  the  outputs  of  higher-level 
fusion  systems. 

Establishing  and  maintaining  context  is  a  fundamental 
requirement  for  information  fusion  ([15]).  Given 
incomplete  information  provided  by  sensors,  context  is  a 
necessary  consideration  for  object  detection  and 
classification.  Context  is  even  more  important  when 
attempting  to  understand  (and  act  on)  the  behavior  of 
objects  in  the  environment,  as  it  depends  on  abstractions 
that  can  not  be  directly  measured.  Failure  to  recognize  this 
can  inadvertently  bias  the  interpretation  of  a  situation  - 
with  potentially  tragic  consequences2. 

Context  is  just  as  important  for  conveying  information  to 
the  users  of  a  command  and  control  (C2)  system.  If 
context  is  not  clearly  established,  users’  will  need  to 
reconstruct  it  from  tacit  knowledge,  background 
information,  and  so  their  interpretation  of  a  COP  will 
depend  on  their  current  roles  and  situations.  By  clearly 
establishing  context,  users  can  rapidly  assimilate  the 
information  provided  and  have  a  greater  likelihood  of:  a) 


2  For  example,  the  NATO  attack  on  the  Djakovica  refugee 
convoy,  cited  in  “Final  Report  to  the  Prosecutor  by  the 
Committee  Established  to  Review  the  NATO  Bombing 
Campaign  Against  the  Federal  Republic  of  Yugoslavia”, 
http://www.un.org/icty/pressreal/nato06 1 300.htm 
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correctly  interpreting  the  situation,  and  b)  sharing  a 
common  interpretation  with  other  users. 

Sensor  fusion  can  be  thought  of  as  establishing  the  data 
about  the  entities  in  the  environment,  while  higher-level 
fusion  is  about  establishing  the  story  behind  the  data. 
Thus,  achieving  situational  awareness  for  the  users  of  an 
information  fusion  system  is  akin  to  storytelling  ([16]), 
which  provides  a  compelling  mechanism  for  describing 
complex  and  contextually  sensitive  relationships. 
Television  news  services  provide  a  highly  successful 
example  of  storytelling  which  incorporates  multimedia  to 
convey  situation  awareness  about  complex  relationships 
in  a  local  and  global  context.  They  establish  context 
through  the  use  of  multimedia  content  such  as  imagery, 
video,  and  graphics,  and  use  narrative  to  assist 
comprehension  of  a  situation  and  its  consequences. 

The  daily  briefing  in  military  command  centers  fulfills 
this  role,  but  the  scope  and  timeliness  of  these  briefings  is 
predicated  on  the  production  process.  A  better  model 
would  be  to  provide  television-style  multimedia 
‘briefings’  (or  ‘multimedia  narrative’)  on  demand  for  any 
situation  as  it  develops,  yet  it  is  not  practical  to  have  a 
television  production  team  assembling  briefings  on  this 
scale  -  an  automated  system  is  needed.  Storytelling  is 
generally  considered  to  be  a  uniquely  human  ability,  but 
there  is  a  growing  body  of  work  ([17])  that  is 
demonstrating  how  real-time  animated  characters  can 
provide  training  outcomes  similar  to  humans. 


Figure  12:  One  of  DSTO’s  Virtual  Advisers 

Real-time  animated  characters,  dubbed  ‘virtual  advisers’, 
have  been  developed  by  Australia’s  Defence  Science  & 
Technology  Organisation  (DSTO)  to  act  as  automated 
storytellers  that  provide  multimedia  narrative  on  demand. 
The  constrained  format  needed  for  military- style  briefings 
provides  a  niche  well-suited  to  the  current  technological 
limitations.  Virtual  advisers  can  interact  with  users  using 
natural  language  and  text-to-speech  technologies,  and 
present  multimedia  content.  Virtual  advisers  can  provide 


additional  context  such  as  importance,  confidence,  and 
urgency  through  non-verbal  cues.  Appearance,  facial 
expressions,  gestures,  behavior  patterns,  and  voice 
prosody  can  all  be  used  to  provide  contextual  cues  to  the 
users.  Incomplete  information  can  be  conveyed  by 
selecting  different  multimedia  elements  to  represent 
different  levels  of  abstraction.  Trust  is  another  important 
factor  that  influences  user  engagement  and  confidence  in 
the  information  presented  ([18]).  It  is  important  to  manage 
the  relationship  between  the  users  and  the  virtual  adviser 
so  that  context  can  be  conveyed  without  disrupting  the 
users’  trust  in  the  virtual  adviser. 


Figure  13:  The  Virtual  Battlespace  smoothly  transitions  to 
and  from  a  ’dots-on-maps'  display  to  a  virtual  reality  scene 
when  appropriate. 


Figure  14:  The  Virtual  Battlespace  provides  a 
photorealistic  3D  geospatial  display  incorporating 
multimedia  annotations. 

Virtual  advisers  provide  a  generic  capability  to  provide 
multimedia  narrative  that  can  be  rendered  on  demand. 
However,  for  military  situation  awareness  a  capability  for 
displaying  geospatial  context  is  also  required.  A  “virtual 
battlespace”  capability  provides  this.  It  is  a  photorealistic 
3D  geospatial  representation  of  the  battlespace,  using 
real-time  track  feeds  and  digital  terrain,  imagery,  and 
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maps  provided  by  web  services.  It  can  convey  to  the  users 
a  much  richer,  and  up-to-date,  representation  of  the  area 
of  operations  than  a  traditional  map.  Sensor  envelopes, 
air  corridors,  and  engagement  zones  can  be  displayed  in 
3D,  and  multimedia  annotations  can  be  dynamically 
added  to  highlight  objects  and  relationships  of  interest, 
and  associate  expansion  information  with  the  entities  in 
the  scene.  This  provides  a  rich  capability  for  multimedia 
narrative  when  orchestrated  with  a  virtual  adviser. 

Correction  of  errors,  or  content  refinement  after  the  fact 
as  more  information  becomes  available,  is  often  difficult. 
This  can  be  facilitated  by  managing  the  users’  trust  in 
particular  content.  ’Untrustworthy’  characters  can  be  used 
to  present  information  where  the  reliability  of  the 
information  is  low,  but  the  risk  associated  with  ignoring  it 
is  high.  Different  media  may  also  have  implicit  levels  of 
trust  associated  with  them:  for  example,  text  is  considered 
a  more  reliable  source  than  a  virtual  adviser  ([19]).  By 
selecting  media  with  lower  associated  levels  of  implicit 
trust,  correction  of  the  users’  situation  awareness  when 
more  reliable  information  is  available  can  be  facilitated. 
To  avoid  unwanted  associations  in  the  Virtual 
Battlespace,  one  approach  to  dealing  with  uncertain 
information  is  to  use  models  that  are  not  identified  with 
any  real  system.  For  example,  an  unidentified  submarine 
could  be  represented  by  a  fictitious  model  that  bears  no 
resemblance  to  any  existing  submarine,  as  shown  in 
Figure  15  below. 


Figure  15:  An  unidentified  submarine  may  be  represented 
by  a  fictitious  model  to  avoid  unwanted  associations. 

The  effective  use  of  multimedia  narrative  as  a  Higher- 
COP  depends  on  the  ability  of  a  computer  system  to 
generate  and  assemble  multimedia  components  to  convey 
a  coherent,  informative,  and  interesting,  message  to  the 
users.  This  requires,  in  some  sense,  the  ability  to  emulate 
human  storytelling  expertise,  and  capture  the  ’’director’s 
art”.  This  is  a  challenging  requirement,  but  it  is  important 
to  note  that  the  aim  of  this  work  is  only  to  provide  some 
capability  to  enhance  situation  awareness.  Basic  templates 
for  layout  of  media,  and  simple  heuristics  for  media 
selection,  could  be  used  to  assemble  a  basic  multimedia 
presentation  to  aid  the  decision  makers’  comprehension  of 
the  situation  and  its  consequences. 

Further  refinement  of  higher-level  information  fusion  and 
more  intelligent  multimedia  selection  and  layout  is 
required  to  realize  an  effective  fully-automated  system  for 


a  Higher  COP.  The  CDIFT  provides  an  ideal 
environment  in  which  to  further  develop  and  trial  these 
capabilities,  with  virtual  adviser  and  virtual  battlespace 
technologies  playing  a  significant  role  as  situational 
awareness  displays. 

In  the  CDIFT,  summary  reports  generated  by  fusion 
applications,  in  particular  the  ‘virtual  Commander  Joint 
Task  Force’  (vCJTF)  agent,  are  published  over  the  JTFIG. 
A  number  of  report  subscribers  select  only  those  reports 
relevant  to  the  particular  role  of  the  site,  priority  and 
context.  Based  on  the  information  content,  context,  and 
user  preferences,  these  reports  can  be  rendered  in  a 
number  of  ways:  as  text  output;  presented  by  the  virtual 
adviser;  or  displayed  in  the  virtual  battlespace.  Currently 
the  content  presented  in  this  way  includes:  simple 
summaries  of  the  numbers  of  entities  observed;  the 
probability  of  association  of  maritime  tracks  with  sea 
lanes;  clustering  of  tracks  based  on  their  kinematics;  and 
prioritized  threat  assessments  of  air  tracks.  This  capability 
will  be  extended  and  refined  further  as  the  CDIFT 
develops  and  more  applications  are  incorporated. 


Figure  16:  Example  of  how  virtual  advisers  and  virtual 
battlespace  are  used  in  the  CDIFT. 

9  Urban  Operations 

One  of  the  best  environments  to  envision  higher-level 
fusion  is  the  urban  environment.  Urban  operations  push 
toward  the  limits  -  combining  army,  navy,  air,  joint  and 
civil  powers  together,  and  ranging  from  the  strategic  to 
the  tactical  levels  ([20]).  Global  urbanization  over  the 
next  20  years  will  create  an  increasingly  demanding 
operational  environment  for  military  forces  to  operate  in. 
Moreover,  these  areas  will  take  on  importance  too,  with 
future  opponents  choosing  to  fight  in  urban  areas  to  offset 
tactical  advantages  of  more  sophisticated  forces.  Simple 
avoidance  of  urban  areas  because  of  challenges  created  by 
a  multi-dimensional  battlespace,  and  the  presence  of  non- 
combatants  and  complex  infrastructures,  will  not  be  an 
option  in  many  cases.  An  Army  must  be  capable  of 
fighting  and  winning  throughout  the  range  of  conflict  and 
in  the  environment  where  decisive  action  is  required. 
Any  force  that  cannot  operate  effectively  in  both  urban 
and  expanded  battlespaces  will  be  severely  restricted  in  its 
future  responsiveness  ([21]).  Historical  and  contemporary 
operations  implying  urban  environments  are  considered  as 
complex  if  not  extreme  cases  (e.g.  Stalingrad,  Hue, 
Mogadishu,  Grozny,  Bagdad,  Kandahar).  They  implied  or 
imply  a  wide  range  of  resources  from  the  whole  spectrum 
of  defense  and  civil  services  (plus  NGO,  media,  etc.)  and 
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interactions  at  all  levels  of  command.  Even  the  study  of 
counterinsurgency  is  closely  linked  with  urban  operations 
since  they  generally  share  the  same  theatre  of  operations. 

The  TTCP  panel  on  Information  Fusion  is  currently 
working  on  an  extension  of  its  basic  scenario  to  include 
urban  operations  ([22]).  The  first  step  has  been  to  include 
-  almost  integrally  -  an  existing  scenario  developed  for  a 
Kingston  Staff  College  (Canada)  exercise  held  in  2002 
entitled  “Urban  Challenge  2025”  ([21]).  A  NATO 
Research  and  Technology  (RTO)  study  afterward 
enriched  it  with  tactical-level  vignettes  ([23], [24]).  To 
define  vignette  enrichments,  military  personnel  and 
scientists  wargamed  the  original  “Urban  Challenge” 
scenario  in  more  specific  contexts.  These  contexts  were 
limited  to  three  mission  types:  crisis  response  operation, 
defensive  operation,  and  offensive  operation.  All  of  these 
developed  and  played  from  a  command  and  control  (C2) 
point  of  view. 

There  is  a  work  in  progress  to  include  this  enriched  urban 
scenario  within  the  panel  basic  scenario  story.  This  will 
most  likely  be  translated  in  an  ethnic  based  urban  conflict 
within  Whiteland  implying  Greyland  immigrants’ 
descendents.  This  group  would  require  more  autonomy 
for  their  province.  The  troubles  would  occur  within  urban 
theaters. 

New  iterations  of  the  urban  vignettes  are  already 
forecasted.  One  proposal  is  to  include  two  scenarios 
defined  by  the  Joint  Forces  Command  -  Urban  Operations 
Office  (US)  ([25]).  The  first  is  entitled 
“Counterinsurgency  in  Port  Fewis,  2015-2021”.  It  covers 
a  domestic  counterinsurgency  context  resulting  in 
bacterial  meningitis  and  influenza  outbreaks.  The  second 
is  entitled  “The  attack  on  Qabus  2027”.  It  covers  a  foreign 
urban  operation  context. 

Finally,  parallel  work  is  currently  performed  to  identify 
significant  urban  vignettes  at  the  tactical  level  to  act 
specifically  as  testbeds  for  higher-level  fusion  discussions 
and  concepts  validation.  Drafted  vignettes  are  up  to  now 
characterized  by  the  large  spectrum  of  information  fusion 
possibilities  they  offer  (e.g.  from  signal  recognition  to 
multi-dimension  situation  assessment). 

10  CSAR 

The  combat  search  and  rescue  (CSAR)  mission  vignette 
developed  at  DRDC  Valcartier  could  be  easily  integrated 
to  CDIFT.  This  vignette  was  placed  into  a  fictitious 
exercise  scenario  -  Final  Fance- Atlantis,  which  was 
borrowed  from  the  Canadian  Forces  Command  and  Staff 
College  (CFCSC).  The  actions  of  each  airborne  tactical 
platform,  as  well  as  the  elements  of  Combined  Air 
Operations  Center  (CAOC)  participating  in  the  CSAR 
mission,  were  described  in  detail  along  the  sequence  of 


events  that  would  occur  during  the  conduct  of  a  CSAR 
mission.  A  particular  emphasis  was  placed  on  the 
methods,  techniques  and  sensors  used  in  compiling  and 
exploiting  both  a  recognized  air  picture  (RAP)  and 
recognized  ground  picture  (RGP)  using  airborne  assets. 
Although  separate  elements  of  the  same  mission,  many  of 
the  RGP  assets  are  airborne  and,  as  such,  are  part  of  the 
RAP.  Therefore,  any  movement  of  air  assets  to  enhance 
the  RGP  would  have  a  concomitant  effect  on  the  RAP.  A 
synchronization  matrix  of  “friendly”  actions  and 
communications  (directive  and  informative)  was  created, 
first,  given  the  ideal  situation  with  no  enemy  action  and 
then  reflecting  the  actions  and  decisions  of  three  (3)  of  the 
following  unpredictable  events: 

a.  Event  1  -  inability  to  locate  the  enemy  ground  positions 
due  to  cloud  and  terrain; 

b.  Event  2  -  enemy  attack  helicopters  appear  in  the  CSAR 
area;  and 

c.  Event  3  -  enemy  SAM  systems  in  the  enemy  rear  area 
detects  the  CF-18  attack  mission. 

Although  CSAR,  like  SAR,  is  considered  a  single 
mission,  it  is  composed  of  two  (2)  separate  and  distinct 
phases  -  the  search  phase  and  the  rescue  phase.  In  a 
peacetime  SAR  scenario,  often  the  search  phase  is 
coincidental  with  the  rescue  phase.  Aircraft,  normally 
SAR  capable,  will  be  dispatched  to  fly  deliberate  search 
patterns  over  the  suspected  area  of  concern.  Once  the  lost 
person(s)  is/are  detected,  like  a  floundering  ship  at  sea, 
and  the  search  aircraft  is  capable,  like  a  SAR  helicopter, 
the  rescue  takes  place  immediately.  However,  in  the  same 
example,  if  the  search  aircraft  was  fixed-wing,  then,  with 
the  search  phase  complete,  the  rescue  mission  would  be 
launched  separately. 

The  same  concept  holds  true  for  the  CSAR  mission  except 
that,  due  to  the  threat  posed  by  conducting  a  rescue 
operation  in  enemy  territory,  the  search  and  rescue  phases 
of  the  mission  are  quite  distinct  and  often  are  separated  by 
days.  There  are  numerous  methods  of  searching  for 
downed  crews  or  personnel  lost  in  enemy  territory.  As 
expected,  the  technique  of  flying  a  predictable  search 
pattern  over  enemy  territory  would  be  fraught  with  risk. 
Combat  search  techniques  would  include: 

a.  location  by  electronic  means  using  emergency  locator 
transmitter  (EFT)  signals  emanating  from  downed 
aircraft.  Those  signals  can  be  triangulated  by 
satellites  like  SARSAT  or  by  aircraft  flying  in 
friendly  territory; 

b.  location  by  radio/secure  radio  transmissions  from  the 
downed  crew(s)  using  escape  and  evasion  radios. 
These  transmissions  can  be  voice,  which  could  pass 
exact  GPS  location  or  a  homing  signal  for 
triangulation.  Again  these  search  techniques  could 
all  occur  over  friendly  territory;  and 

c.  fly  tactically  supported  reconnaissance  assets  (aircraft 
or  UAVs)  through  the  suspected  target  location  to 
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visually  spot  the  crew(s).  This  is  normally  not  done 
due  to  the  risk  of  losing  another  asset  and  because  the 
personnel  on  the  ground  have  no  way  of  knowing  that 
the  reconnaissance  mission  is  friendly  and  thus  will 
likely  be  avoiding/evading  detection. 

On  the  second  day  following  the  commencement  of  the 
Alliance  joint  operations  to  secure  Blueland  territory  and 
expel  any  Coalition  invasion  forces,  a  UK  Royal  Air 
Force  (RAF)  Tornado  call-sign  HAWK27,  conducting  an 
electronic  countermeasures  and  reconnaissance  (ECR) 
mission,  was  shot  down  over  the  Celtic  Straits  by  a 
surface-to-air  missile  (SAM)  at  1608  hours.  The  crew  did 
not  report  any  radar  activity,  so  it  was  believed  that  the 
missile  was  either  an  optically  launched  SA  8  or  shoulder 
launched  SA  14.  Both  systems  had  been  reported  in  the 
area  as  part  of  the  Coalition  Airborne  Regiment  that 
invaded  the  Blueland  portion  of  the  Camrien  Peninsula  at 
the  outset  of  hostilities. 

The  Tornado  aircraft’s  wingman  reported  that  both 
aircrew  had  ejected  safely  and  that  the  downed  crew  had 
reported  no  injuries  via  secure  communication  using  their 
survival  radio.  Shortly  after  the  location  of  the  downed 
crew  was  known,  a  CH-124  Sea  King  helicopter  from 
Wahhabe  Airbase,  with  a  crew  of  five,  was  sent  to  recover 
and  evacuate  the  Tornado  aircrew.  At  approximately 
1800  hours,  in  the  process  of  extraction  of  the  downed 
Tornado  crew,  the  Sea  King  crashed.  The  cause  of  the 
crash  is  attributed  to  mechanical  failure.  Two  (2)  members 
of  the  Sea  King  crew  sustained  non-life-threatening 
injuries  that  have  limited  their  mobility  on  foot.  The 
crash  site,  illustrated  in  Figure  17,  and  location  of  both 
crews  is  5650N  2740W,  which  is  approximately  60  nm 
north  of  the  Brownland  town  of  Amitava  on  the  Camrien 
Peninsula. 


Figure  17:  Tornado  and  Sea  King  Crash  Site. 

The  situation  was  forwarded  from  the  Combined  Air 
Operations  Centre  (CAOC)  to  the  Air  Component 
Commander  (ACC)  for  consideration  in  the  Joint  Force 


Commander’s  (JFC)  force  allocation  for  the  upcoming 
planning  period. 

Finally,  data  sets,  describing  the  individual  characteristics 
of  each  of  the  friendly  and  enemy  airborne  assets  and 
ground-based  SAM  systems  found  in  the  CSAR  scenario 
can  be  provided  to  assist  in  follow-on  CDIFT  simulation 
and/or  ontological  modelling. 

11  Conclusion 

This  paper  outlines  a  collaborative  program  of  work  in 
higher-level  fusion  being  conducted  between  Australia, 
Canada,  the  United  States  and  the  United  Kingdom  under 
the  auspices  of  a  TTCP  panel  on  Information  Fusion.  The 
paper  shows  how  work  from  the  four  nations  has  been 
integrated  through  a  common  scenario,  which  the  panel 
proposes  to  subsequently  offer  as  a  benchmarking 
scenario  for  the  higher-level  fusion  community  more 
broadly. 
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